Image acquisition using time-multiplexed chromatic illumination

ABSTRACT

An imaging system includes an image sensor having an array of sensor pixels, a multi-chromatic illuminator adapted to independently generate a plurality of illumination colors each having a different finite spectral range, and a controller coupled to the image sensor and to the multi-chromatic illuminator. The controller includes logic to time-multiplexing the multi-chromatic illuminator between the illumination colors. Each of the illumination colors independently illuminates for at least a corresponding one of non-overlapping durations of illumination of the illumination colors. The controller includes further logic to acquire chromatic sub-images with the image sensor and combine the chromatic sub-images into a composite color image. Each of the chromatic sub-images is acquired during a different one of the non-overlapping durations of illumination.

TECHNICAL FIELD

This disclosure relates generally to color image sensors, and inparticular but not exclusively, relates to super-resolution demosaicingor high dynamic range imaging.

BACKGROUND INFORMATION

Demosaicing reconstructs a full color image from incomplete colorsamples output from an image sensor that is overlaid by a color filterarray (CFA). The demosaicing process sacrifices spatial resolution dueto the need to perform interpolation in order to obtain the red, greenand blue (R, G, B) color data for each image pixel. Both charge coupleddevice (CCD) and complementary metal-oxide-semiconductor (CMOS) colorimage sensors suffer reduced spatial resolution due to the demosaicingprocess. The reduction in resolution results is a direct result of thefact that each sensor pixel in the image sensor has only one colorfilter (R or G or B), and the CFA is typically structured in groups offour sensor pixels, where one of the sensor pixels has a blue filter,another has a red filter, while the remaining two have green filters, asseen in FIG. 1. This common pattern is referred to as a Bayer pattern.Thus, for any given image sensor array, one-half of the sensor pixelshave a green color filter, one-quarter of the sensor pixels have a bluecolor filter, while one-quarter of the sensor pixels have a red colorfilter.

Since virtually all color image sensors employ the Bayer pattern CFAstructure detailed above, and consequently interpolate between nearestlike-colored pixels in order to obtain R, G, B data for each imagepixel, it follows that using a color image sensor inherently limits theNyquist spatial frequency of the blue and red channels to one-half theNyquist frequency of an identical camera whose color filters wereremoved. Similarly, the green channel's Nyquist frequency is reduced by1/√{square root over (2)}. This is the price of color.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive embodiments of the invention aredescribed with reference to the following figures, wherein likereference numerals refer to like parts throughout the various viewsunless otherwise specified. Not all instances of an element arenecessarily labeled so as not to clutter the drawings where appropriate.The drawings are not necessarily to scale, emphasis instead being placedupon illustrating the principles being described.

FIG. 1 (PRIOR ART) illustrates a conventional Bayer pattern color filterarray including red, green, and blue filter elements that overlay animage sensor array.

FIG. 2A is a functional block diagram illustrating an imaging systemthat time-multiplexes a multi-chromatic illuminator to acquire chromaticsub-images, in accordance with an embodiment of the disclosure.

FIG. 2B illustrates the combination of chromatic sub-images into acomposite color image, which may be further optimized, in accordancewith an embodiment of the disclosure.

FIGS. 3A, 3B, and 3C illustrate example implementations of amulti-chromatic illuminator, in accordance with embodiments of thedisclosure.

FIG. 4 is a chart illustrating wavelength dependent quantum efficienciesof red, green, and blue color filter elements, in accordance with anembodiment of the disclosure.

FIG. 5 illustrates a color filter array including a customized repeatingpattern scheme, in accordance with an embodiment of the disclosure.

FIG. 6 illustrates a color filter array including a customized repeatingpattern scheme, in accordance with another embodiment of the disclosure.

FIG. 7 is a flow chart illustrating a process for acquiring a colorimage by time-multiplexing a multi-chromatic illuminator, in accordancewith an embodiment of the disclosure.

DETAILED DESCRIPTION

Embodiments of a system, apparatus, and method of operation for a camerasystem that acquires color images using time multiplexed chromaticillumination to generate super-resolution color images and/or highdefinition range (HDR) color images are described herein. In thefollowing description numerous specific details are set forth to providea thorough understanding of the embodiments. One skilled in the relevantart will recognize, however, that the techniques described herein can bepracticed without one or more of the specific details, or with othermethods, components, materials, etc. In other instances, well-knownstructures, materials, or operations are not shown or described indetail to avoid obscuring certain aspects.

Reference throughout this specification to “one embodiment” or “anembodiment” means that a particular feature, structure, orcharacteristic described in connection with the embodiment is includedin at least one embodiment of the present invention. Thus, theappearances of the phrases “in one embodiment” or “in an embodiment” invarious places throughout this specification are not necessarily allreferring to the same embodiment. Furthermore, the particular features,structures, or characteristics may be combined in any suitable manner inone or more embodiments.

The field of super-resolution includes various conventional algorithmswhose aim is to increase the resolution of a color image via thedemosaicing/interpolation process. These conventional algorithms aretypically computational only, attempting to use either a model or apriori information to extract a higher resolution image from the regularresolution image captured by a color image sensor array. In contrast,embodiments described herein use a sequential multi-chromaticillumination scheme coupled with knowledge of the quantum efficiencies(QEs) of the sensor pixels and color filter array (CFA) to generatesuper resolution images and/or HDR images from a color image sensor.

FIG. 2A illustrates the components of an imaging system 200 capable ofacquiring color images using time multiplexed multi-chromaticillumination to generate super-resolution color images and/or HDR colorimages. The illustrated embodiment of system 200 includes an imagesensor 205 including a sensor array 207 overlaid by a CFA 210, amulti-chromatic illuminator 215, a controller 220, and a lens 225.

Sensor array 207 may be implemented using a variety of image sensortechnologies including CCD sensors, CMOS sensors, or otherwise.Controller 220 orchestrates the operation of the other functionalcomponents and may include logic implemented in hardware (e.g.,application specific integrated circuit, field programmable gate array,etc.), logic instructions implemented in software/firmware and executedby a general purpose microcontroller, or implemented with a combinationof hardware and software logic. CFA 210 may be implemented as a Bayerpattern CFA having red (R), green (G), and blue (B) filter elements, orother CFA patterns using other color filter elements (e.g., CYYM, CYGM,RGBW, etc.).

Multi-chromatic illuminator 215 is a multispectral composite lightsource capable of sequentially generating different illumination colorsof finite spectral range or linewidth. For visible spectrum colorimages, in order to provide sufficient color accuracy in the outputcolor image, multi-chromatic illuminator 215 may have a first colorsource, such as a blue source with some energy in the 400-460 nmwavelength range, a second color source, such as a green source withsome energy in the 460 nm-560 nm wavelength range, and a third colorsource, such as a red source with some energy in the 620 nm-700 nmrange. Of course, if image sensor 205 extends beyond the visiblespectrum or operates entirely outside the visible spectrum, thenmulti-chromatic illuminator 215 may generate illumination colors inother wavelength bands and may even include more or less than threeillumination sources working across more or less distinct finitespectral ranges.

FIGS. 3A-3C illustrate different examples for implementingmulti-chromatic illuminator 215, in various embodiments. For example,FIG. 3A illustrates a multi-chromatic illuminator 301 formed from threediscrete illumination sources 305A, 305B, and 205C each tuned forgenerating illumination light having a different illumination color. Forexample, illuminator source 305A outputs red illumination light,illuminator source 305B outputs green illumination light, andilluminator source 305C outputs blue illumination light. Of course,other color and wavelength bands or spectral ranges may be implemented.FIG. 3B illustrates a multi-chromatic illuminator 302 implemented usinga broadband illuminator source 310 (e.g., generating white light)overlaid by a tunable/dynamic set of bandlimited filters 315A-C. Filters315A-C may be moved, or otherwise selected, in a timeshared manner. FIG.3C illustrates a multi-chromatic illuminator 303 implemented using atunable light source 320 capable of selectively generating multipledistinct, bandlimited illumination colors.

Turning to FIG. 2B, imaging system 200 is capable of capturingsuper-resolution and/or HDR images using a technique that acquiresmultiple chromatic sub-images 230A-C each illuminated by a differentillumination color by multi-chromatic illuminator 215 duringnon-overlapping temporal durations. In various embodiments, thesechromatic sub-images may be acquired at 3× the desired frame-rate.Multiple chromatic sub-images 230A-C are extracted from image sensor 205as high-resolution images by leveraging knowledge of the spectral QE ofimage sensor 205. For example, FIG. 4 illustrates a typical quantumefficiency of red, green, and blue color filter elements in a CFA. Asillustrated, the wavelength dependent QE can be characterized. Thiswavelength dependent QE can be leveraged to extract red and blue colorimage values from a sensor pixel covered by a green filter, extractgreen and blue color image values from a sensor pixel covered by a redfilter, and extract red and green color image values from a sensor pixelcovered by a blue filter when the illumination colors aretime-multiplexed for at least some non-overlapping durations. The QE isoften available from image sensor manufactures, but can also bemeasured. The QEs of the three color filter types in CFA 210 of atypical Bayer pattern filter will be referred to as QE^((c))(λ), wherec=r, g, or b.

Returning to FIG. 3A (as a demonstrative example), the spectralirradiance (e.g., watts per square meter) of each illuminator source 305of multi-chromatic illuminator 301 is denoted as follows: sources whosespectral irradiance lies within the blue color filter sensitivity rangeare denoted by S_((b)) ⁽¹⁾(λ), S_((b)) ⁽²⁾(λ) . . . S_((b)) ^((N))(λ),where the superscript denotes the number of the source. Similarly,sources whose spectral irradiance lies within the green color filtersensitivity range will be denoted as S_((g)) ⁽¹⁾(λ), S_((g)) ⁽²⁾(λ) . .. S_((g)) ^((N))(λ), and sources whose spectral irradiance lies withinthe red color filter sensitivity range will be denoted as S_((r))⁽¹⁾(λ), S_((r)) ⁽²⁾(λ) . . . S_((r)) ^((N))(λ).

Controller 220 operates to generate composite color image 240 fromchromatic sub-images 230 as follows. First, chromatic sub-image 230A iscaptured with all blue sources on, denoted by I_(b)[n_(x), n_(y)]. Then,we define I_(b) ^((b))[n_(x), n_(y)], the image obtained from i_(b) byzeroing out the intensity values for all green and red color filters.n_(x) and n_(y) are indexes running over all the pixel columns and rowsof the image sensor. Note, the superscript of I corresponds to the Bayerfilter color and the subscript corresponds to the illumination sourcecolor, I_(source color) ^((filter color)).

Second, we capture chromatic sub-image 230B with all green sources on,denoted by I_(g)[n_(x), n_(y)]. Then, we define I_(b) ^((g))[n_(x),n_(y)], the image obtained from I_(g) by zeroing out the intensityvalues for all blue and red color filters. n_(x) and n_(y) are indexesrunning over all the pixel columns and rows of the image sensor.

Third, we capture chromatic sub-image 230C with all red sources on,denoted by I_(r)[n_(x), n_(y)]. Then, we define I_(r) ^((r))[n_(x),n_(y)], the image obtained from I, by zeroing out the intensity valuesfor all green and blue color filters. n_(x) and n_(y) are indexesrunning over all the pixel columns and rows of the image sensor.

With the three chromatic sub-images 230 acquired, composite color image240 is created based upon a combination of the three chromaticsub-images 230 as follows (superscript HR stands for high resolution):

${I^{({{HR},b})}\left\lbrack {n_{x},n_{y}} \right\rbrack} = {\frac{I_{b}^{(b)}\left\lbrack {n_{x},n_{y}} \right\rbrack}{\sum\limits_{i = 1}^{N}{\int{d\; \lambda \; {S_{b}^{(i)}(\lambda)}{{QE}^{(b)}(\lambda)}}}} + \frac{I_{b}^{(g)}\left\lbrack {n_{x},n_{y}} \right\rbrack}{\sum\limits_{i = 1}^{N}{\int{d\; \lambda \; {S_{b}^{(i)}(\lambda)}{{QE}^{(g)}(\lambda)}}}} + \frac{I_{b}^{(r)}\left\lbrack {n_{x},n_{y}} \right\rbrack}{\sum\limits_{i = 1}^{N}{\int{d\; \lambda \; {S_{b}^{(i)}(\lambda)}{{QE}^{(r)}(\lambda)}}}}}$${I^{({{HR},g})}\left\lbrack {n_{x},n_{y}} \right\rbrack} = {\frac{I_{g}^{(b)}\left\lbrack {n_{x},n_{y}} \right\rbrack}{\sum\limits_{i = 1}^{N}{\int{d\; \lambda \; {S_{g}^{(i)}(\lambda)}{{QE}^{(b)}(\lambda)}}}} + \frac{I_{g}^{(g)}\left\lbrack {n_{x},n_{y}} \right\rbrack}{\sum\limits_{i = 1}^{N}{\int{d\; \lambda \; {S_{g}^{(i)}(\lambda)}{{QE}^{(g)}(\lambda)}}}} + \frac{I_{b}^{(r)}\left\lbrack {n_{x},n_{y}} \right\rbrack}{\sum\limits_{i = 1}^{N}{\int{d\; \lambda \; {S_{g}^{(i)}(\lambda)}{{QE}^{(r)}(\lambda)}}}}}$${I^{({{HR},r})}\left\lbrack {n_{x},n_{y}} \right\rbrack} = {\frac{I_{r}^{(b)}\left\lbrack {n_{x},n_{y}} \right\rbrack}{\sum\limits_{i = 1}^{N}{\int{d\; \lambda \; {S_{r}^{(i)}(\lambda)}{{QE}^{(b)}(\lambda)}}}} + \frac{I_{r}^{(g)}\left\lbrack {n_{x},n_{y}} \right\rbrack}{\sum\limits_{i = 1}^{N}{\int{d\; \lambda \; {S_{r}^{(i)}(\lambda)}{{QE}^{(g)}(\lambda)}}}} + {\frac{I_{r}^{(r)}\left\lbrack {n_{x},n_{y}} \right\rbrack}{\sum\limits_{i = 1}^{N}{\int{d\; \lambda \; {S_{r}^{(i)}(\lambda)}{{QE}^{(r)}(\lambda)}}}}.}}$

Finally, we define composite color image 240 as:

I ^((HR))[n _(x) , n _(y)]=(I ^((HR,b))[n _(x) , n _(y)], I ^((HR,g))[n_(x) , n _(y)], I ^((HR,r))[n _(x) , n _(y)]).

At this point, composite color image 240, I^((HR))[n_(x), n_(y)], can beused as a high resolution version of the original full color imageI[n_(x), n_(y)] (which would be a conventional image obtained by turningall of the sources on at once, taking an image, and performing pixeldemosaicing (including pixel interpolation)).

Due to the variation of the QEs of the color filter elements across thevisible spectrum, in addition to having higher spatial resolution thanI[n_(x), n_(y)], I^((HR))[n_(x), n_(y)] also has higher noise levels. Toreduce the noise levels, I^((HR))[n_(x), n_(y)] can be further refinedor optimized using a variety of strategies. Since a preferred imagequality metric relevant in this situation is some combination of thedynamic range of the image and the spatial resolution of the image, anoptimized image is a combination of I^((HR))[n_(x), n_(y)] and I[n_(x),n_(y)] (the conventional full color image). Consequently, compositecolor image 240 can be further refined to generate the optimal image:

I^((Optimal))[n_(x), n_(y)]=K⁽¹⁾[n_(x), n_(y)]*I[n_(x),n_(y)]+K⁽²⁾[n_(x), n_(y)]*I^((HR))[n_(x), n_(y)], where K⁽¹⁾[n_(x),n_(y)] and K⁽²⁾[n_(x), n_(y)] are two-dimensional convolution kernelswith finite spatial support that are solved by optimizing for a meritfunction combining spatial resolution and dynamic range. The symbol *denotes a two-dimensional convolution, i.e.:

${{K^{(1)}\left\lbrack {n_{x},n_{y}} \right\rbrack}*{I\left\lbrack {n_{x},n_{y}} \right\rbrack}} = {\sum\limits_{n_{x}^{\prime} = {- \infty}}^{\infty}{\sum\limits_{n_{y}^{\prime} = {- \infty}}^{\infty}{{K^{(1)}\left\lbrack {n_{x}^{\prime},n_{y}^{\prime}} \right\rbrack}{I\left\lbrack {{n_{x} - n_{x}^{\prime}},{n_{y} - n_{y}^{\prime}}} \right\rbrack}}}}$$\mspace{79mu} {{{K^{(2)}\left\lbrack {n_{x},n_{y}} \right\rbrack}*{I\left\lbrack {n_{x},n_{y}} \right\rbrack}} = {\sum\limits_{n_{x}^{\prime} = {- \infty}}^{\infty}{\sum\limits_{n_{y}^{\prime} = {- \infty}}^{\infty}{{K^{(2)}\left\lbrack {n_{x}^{\prime},n_{y}^{\prime}} \right\rbrack}{I\left\lbrack {{n_{x} - n_{x}^{\prime}},{n_{y} - n_{y}^{\prime}}} \right\rbrack}}}}}$

This optimization can be performed by acquiring the following threeimages:

Image 1: an image whose object space contains, for example, a 50%diffuse reflectance standard;

Image 2: an image whose object space contains a black reflectancestandard (i.e., very low diffuse reflectance); and

Image 3: an image whose object space contains a slanted edge resolutiontarget. Using the images 1 and 2, a dynamic range value can becalculated in a patch of image pixels of a general size (can be forevery single image pixel). Using image 3, the spatial resolution can becalculated. From this, a merit function combining the two metrics can beconstructed. Using this merit function, we can solve for K⁽¹⁾[n_(x),n_(y)], and K⁽²⁾[n_(x), n_(y)]. Depending on the size of the kernelfunctions' spatial support, the number of images required to solve forthe kernel can vary. Accordingly, a conventional full color image 241,acquired with all illumination colors of multi-chromatic illuminator 215simultaneously enabled, can be combined with super resolution compositecolor image 240 via convolution kernels K⁽¹⁾[n_(x), n_(y)] andK⁽²⁾[n_(x), n_(y)] to generate a yet further optimized composite colorimage 242.

Knowledge of the wavelength dependent QE of the color pixels or colorfilters can be used to further optimize the color data extracted fromimage sensor 205. For example, consider a particular sensor pixel inpixel array 207 covered by a red filter type in the Bayer CFA 210.Normally it is used to measure red light. But it can also be used tomeasure blue light (or even green light). This measurement typically hasa high variance because it requires high gain due to the red filterblocking most (but not all) of the incident blue light. However, thereare nearby blue sensor pixels used to measure blue light and these willhave a lower variance due to a lower applied gain. With a priorievidence that these nearby blue sensor pixels should have color that iscorrelated to the blue light entering at the red sensor pixel, thisknowledge can be used to reduce the variance measurement (i.e., reducethe noise). Instead of using the high variance measurement on its own, aweighted combination of the high variance color data and its neighboringlow variance color data is used. Given an objective measure of imagequality (e.g., the sum of the variances for each sensor pixel) theweights to optimize this measure can be chosen. Since a variance of tenis proportional to the power of the signal, the signal power sometimescan be used as an analogous metric. Furthermore, with knowledge of thesample absorption and scattering profile, the weights can be furthertuned.

Generically, the above optimization extracts first color data (e.g., redcolor data) from a sensor pixel overlaid by a filter element in CFA 210of a different color filter type (e.g., green or blue) with reference tothe wavelength dependent QE of the different color filter type. Thisextracted data is referred to as high noise data as it is expected tohave a high noise level compared to extracting red color data from a redcolor sensor pixel (or green data from a green color sensor pixel orblue data from a blue color sensor pixel), which is referred to as lownoise data. Accordingly, high noise data is optimized with reference toadjacent low noise data by taking a weighted combination of the lownoise data with the high noise data.

Knowledge of the wavelength dependent QE of the color sensor pixels orcolor filter types in CFA 210 can be leveraged to acquire HDR colorimages as mentioned above. In particular, the wavelength dependent QEsof the color filter types in CFA 210 can be used to scale the imagevalues output from the sensor pixels when combining chromatic sub-images230 into composite color image 240. This optimization is included withinthe equations presented above for I^((HR,b))[n_(x), n_(y)],I^((HR,g))[n_(x), n_(y)], and I^((HR,r))[n_(x), n_(y)] where thedenominator that scales the chromatic sub-images 230A, 230B, and 230Cincludes a different QE^((b)), QE^((g)), and QE^((r)), respectively.

One common challenge in medical imaging, videography, and photography iscapturing HDR scenery. A scene with a high dynamic range refers to onein which there are bright objects, which are often overexposed (white),alongside dark objects, which are often underexposed (black).Conventionally, the challenge of imaging HDR scenery is addressed byimaging the scene multiple times with di□erent exposure times andcompiling the resulting images into a single balanced photo. Since thecolor filters of the Bayer pattern have di□erent attenuation coe□cientsat the various wavelengths (i.e., wavelength dependent QEs of CFA 210),they can be used analogously to di□erent exposure times of a camera.Rather than capturing three di□erent exposures sequentially, this HDRimaging scheme allows for all three exposures to be capturedsimultaneously by the red, green and blue filtered sensor pixels. Forexample, if the scene is illuminated by the a red source and the sensorpixel under the matched red color filter saturates, then the sensorpixel under the blue color filter that heavily attenuates red lightcould still be used to capture the image without over saturating theunderlying sensor pixel. The value under the blue sensor pixel may benoisy (high noise data) but it provides a relative intensity di□erenceto other non-saturated pixels. The weighting kernel described previouslycan be modified to account for oversaturated pixels, thereby improvingthe dynamic range of the imaged scene in addition to improving theresolution of the image extracted from image sensor 205.

Accordingly, chromatic sub-images 230 may be combined to generate a HDRcomposite color image 240. This HDR combining technique includesidentifying a saturated image value output from a saturated sensor pixeloverlaid by a filter element of a first color type. The saturated imagevalue is then adjusted based upon another image value extracted from anadjacent sensor pixel that is overlaid by a filter element of a secondcolor type (different from the first color type). The other image valuefrom the adjacent sensor pixel is determined with reference to thewavelength dependent QE of the second color filter type.

Exposing a scene sequentially to a red, then green, then blue lightsources (or other illumination color combinations) and repeating thatorder is a logical approach, but other orderings could be beneficial inthe event that there is object motion between exposures. For example, ifall of the blue objects during the blue light exposure happen to alignwith red and green filtered sensor pixels, then the overall noise of theimage will be high. In this scenario, it may make sense to expose thescene to the blue source again in hopes of the pixels align di□erentlyor provide the ability to average multiple blue exposures together.Total power of a given chromatic sub-image 230, obtained by summing thepower of all image values in the given chromatic sub-image 230 beforeapplying gain, is one such metric that could be used to determinewhether the image is of low or high quality. In one embodiment, thesummed total power is compared against a threshold total power value. Ifthe total power associated with a given chromatic sub-image 230 fallsbelow the threshold total power value, then the given chromaticsub-image 230 is reacquired with the associated illumination color.Additionally, since there is twice the number of green pixels to red andblue pixels in a typical Bayer pattern, it may make sense to have twicethe number of red and blue source exposures in order to balance out thisdiscrepancy.

Natural movement of the scene allows for noise reduction throughtemporal averaging as well. If the red source illuminates an object thatmaps to a blue filtered pixel, the noise from gain will be high. Whenthe scene moves, that same object may now map to a red filtered pixelwhere the noise is low. By tracking the movement of a given scene andwhether an object is over a high or low noise pixel, a temporalwindowing function can be applied to the object that adaptively weightsthe kernels based on prior measurements as the object moves between ahigh noise pixel and a low noise pixel.

There are additional schemes that can take advantage of time-multiplexedmulti-chromatic illumination, assuming that CFA 210 is not a traditionalBayer color filter array. FIG. 5 illustrates a first example custom CFA500 that uses a different repeating pattern than a conventional Bayerpattern. In particular, CFA 500 includes a repeating pattern scheme thatreplaces the second green color filter in the Bayer pattern with afourth color filter (e.g. yellow) that is matched to a fourth colorsource in multi-chromatic illuminator 215. The yellow channel o□ers acompromise in noise between the red and green channels and equalizes thespatial color distribution to 1/4 red, 1/4 green, 1/4 blue, and 1/4yellow. Alternatively, if no color filter is used in the fourth spot,then a filterless channel would o□er a robust, low-noise signal whichcould be used to adjust the weights of the adjacent pixels during thethree RGB exposures. Of course, the fourth pixel spot may be overlaid bya color filter having a non-RGB color other than yellow or clear.

FIG. 6 illustrates a example custom CFA 600 that uses yet anotherdifferent repeating pattern scheme, in accordance with anotherembodiment. The scheme illustrated in FIG. 6. alternates the fourthsensor pixel spot (i.e., the second green channel filter location of aBayer pattern) between a rotating color (e.g., rotating between red,green, and blue color filters). An example distribution is shown in FIG.6. This approach creates redundant RGB sensor pixels every six pixelslaterally. This o□ers similar advantages to the original 1/4 red, 1/2green, 1/4 blue distribution of a Bayer pattern but makes the split 1/3red, 1/3 green, 1/3 blue, thereby improving the robustness to noise.

Yet a third alternative scheme removes the color filters entirely,making the sensor monochromatic, but still benefits fromweighting/scaling the sub-images by the wavelength specific QE of themonochromatic sensor array. In this version, sequential exposure to thethree colored sources is synchronized with the camera image acquisition.The pixels would no longer need to be amplified with gain, therebyeliminating amplified noise. Additionally, the absence of a filterallows for 100% of the power to be captured by the sensor, o□ering anadvantage in power throughput.

FIG. 7 is a flow chart illustrating a process 700 for generatingcomposite color image 240, in accordance with an embodiment of thedisclosure. The order in which some or all of the process blocks appearin process 700 should not be deemed limiting. Rather, one of ordinaryskill in the art having the benefit of the present disclosure willunderstand that some of the process blocks may be executed in a varietyof orders not illustrated, or even in parallel.

In a process block 705, multi-chromatic illuminator 215 istime-multiplexed between different illumination colors. Each of theillumination colors is time-multiplexed to independently illuminate forat least a corresponding one of non-overlapping durations ofillumination of the illumination colors. As such, the differentillumination colors may overlap for some period during transitionsbetween the sequential illumination colors; however, the illuminationscheme should include at least some period of non-overlapping durationduring which image acquisition occurs.

In a process block 710, chromatic sub-images 230 are acquired with imagesensor 205. As mentioned, each chromatic sub-image 230 is acquiredduring a different one of the non-overlapping durations of illumination.

In a process block 715, the chromatic sub-images 230 are combined intocomposite color image 240 using the techniques described above. Finally,in a process block 720, composite color image 240 may optionally befurther optimized using any one or more of the above describedoptimizations.

The processes explained above are described in terms of computersoftware and hardware. The techniques described may constitutemachine-executable instructions embodied within a tangible ornon-transitory machine (e.g., computer) readable storage medium, thatwhen executed by a machine will cause the machine to perform theoperations described. Additionally, the processes may be embodied withinhardware, such as an application specific integrated circuit (“ASIC”) orotherwise.

A tangible machine-readable storage medium includes any mechanism thatprovides (i.e., stores) information in a non-transitory form accessibleby a machine (e.g., a computer, network device, personal digitalassistant, manufacturing tool, any device with a set of one or moreprocessors, etc.). For example, a machine-readable storage mediumincludes recordable/non-recordable media (e.g., read only memory (ROM),random access memory (RAM), magnetic disk storage media, optical storagemedia, flash memory devices, etc.).

The above description of illustrated embodiments of the invention,including what is described in the Abstract, is not intended to beexhaustive or to limit the invention to the precise forms disclosed.While specific embodiments of, and examples for, the invention aredescribed herein for illustrative purposes, various modifications arepossible within the scope of the invention, as those skilled in therelevant art will recognize.

These modifications can be made to the invention in light of the abovedetailed description. The terms used in the following claims should notbe construed to limit the invention to the specific embodimentsdisclosed in the specification. Rather, the scope of the invention is tobe determined entirely by the following claims, which are to beconstrued in accordance with established doctrines of claiminterpretation.

What is claimed is:
 1. An imaging system, comprising: an image sensorincluding an array of sensor pixels; a multi-chromatic illuminatoradapted to independently generate a plurality of illumination colorseach having a different finite spectral range; and a controller coupledto the image sensor and to the multi-chromatic illuminator, thecontroller including logic that, when executed by the controller, willcause the imaging system to perform operations comprising:time-multiplexing the multi-chromatic illuminator between theillumination colors, wherein each of the illumination colorsindependently illuminates for at least a corresponding one ofnon-overlapping durations of illumination of the illumination colors;acquiring chromatic sub-images with the image sensor, wherein each ofthe chromatic sub-images is acquired during a different one of thenon-overlapping durations of illumination; and combining the chromaticsub-images into a composite color image.
 2. The imaging system of claim1, further comprising a color filter array (CFA) overlaying the array ofsensor pixels that includes a plurality of different color filter typeseach corresponding to one of the plurality of illumination colors. 3.The imaging system of claim 2, wherein combining the chromaticsub-images into the composite color image comprises: scaling imagevalues output from the sensor pixels based at least in part uponwavelength dependent efficiencies of the color filter types.
 4. Theimaging system of claim 3, wherein the composite color image is a highdynamic range (HDR) image generated based upon the chromatic sub-images,and wherein combining the chromatic sub-images into the HDR imagecomprises: identifying a saturated image value from a saturated sensorpixel overlaid by a filter element of a first color filter type; andadjusting the saturated image value based upon another image valueextracted from another sensor pixel adjacent to the saturated sensorpixel but overlaid by another filter element of a second color filtertype, wherein the other image value is determined with reference to agiven wavelength dependent efficiency of the second color filter type.5. The imaging system of claim 2, wherein the CFA includes a first colorfilter type corresponding to a first illumination color, a second colorfilter type corresponding to a second illumination color, and a thirdcolor filter type corresponding to a third illumination color.
 6. Theimaging system of claim 5, wherein the operations further comprise:extracting first color data corresponding to the first illuminationcolor from a given sensor pixel overlaid by a filter element of thesecond color filter type with reference to a wavelength dependentefficiency of the second color filter type.
 7. The imaging system ofclaim 6, wherein the operations further comprise: optimizing the firstcolor data extracted from the given sensor pixel overlaid by the filterelement of the second color filter type using a weighted combination ofthe first color data from the given sensor pixel and other first colordata output from another sensor pixel adjacent to the given sensor pixelin the array that is overlaid by another filter element of the firstcolor filter type.
 8. The imaging system of claim 5, wherein combiningthe chromatic sub-images into the composite color image comprises:scaling first image values output from the sensor pixels under firstfilter elements of the first color filter type based at least in partupon a first wavelength dependent efficiency of the first color filtertype; scaling second image values output from the sensor pixels undersecond filter elements of the second color filter type based at least inpart upon a second wavelength dependent efficiency of the second colorfilter type; and scaling third image values output from the sensorpixels under third filter elements of the third color filter type basedat least in part upon a third wavelength dependent efficiency of thethird color filter types.
 9. The imaging system of claim 1, wherein theoperations further comprise: acquiring a full color image with all ofthe illumination colors of the multi-chromatic illuminator enabled; andcombining the composite color image with the full color image with oneor more convolution kernels that trade off image characteristics togenerate an optimized composite color image.
 10. The imaging system ofclaim 9, wherein the one or more convolution kernels are two-dimensionalconvolution kernels selected based upon solving a merit function thattrades off the image characteristics including a dynamic range and aspatial resolution.
 11. The imaging system of claim 9, wherein theoperations further comprise: temporally tracking an object moving in thechromatic sub-images; and adaptively adjusting the one or moreconvolution kernels as the object moves between a high noise pixel and alow noise pixel.
 12. The imaging system of claim 1, wherein theoperations further comprise: determining a total power of a givenchromatic sub-image by summing image values of the given chromaticsub-image; comparing the total power to a threshold total power; andreacquiring the given chromatic sub-image when the total power is belowthe threshold total power.
 13. The imaging system of claim 1, whereinthe array of sensor pixels comprises a repeating pattern of color pixelunits, wherein each of the color pixel unit includes: a red sensorpixel, a green sensor pixel, a blue sensor pixel, and a fourth colorsensor pixel having a color that is different than the red sensor pixel,the green sensor pixel, and the blue sensor pixel; or a red sensorpixel, a green sensor pixel, a blue sensor pixel, and a fourth colorsensor pixel having a rotating color that changes across the array ofsensor pixels.
 14. A method for acquiring a color image with an imagesensor having an array of sensor pixels, the method comprising:time-multiplexing a multi-chromatic illuminator between illuminationcolors, wherein each of the illumination colors independentlyilluminates for at least a corresponding one of non-overlappingdurations of illumination of the illumination colors; acquiringchromatic sub-images with the image sensor, wherein each of thechromatic sub-images is acquired during a different one of thenon-overlapping durations of illumination; and combining the chromaticsub-images into a composite color image.
 15. The method of claim 14,wherein the image sensor is overlaid with a color filter array (CFA)that includes a plurality of different color filter types eachcorresponding to one of the plurality of illumination colors.
 16. Themethod of claim 15, wherein combining the chromatic sub-images into thecomposite color image comprises: scaling image values output from sensorpixels of the image sensor based at least in part upon wavelengthdependent efficiencies of the color filter types.
 17. The method ofclaim 16, wherein the composite color image is a high dynamic range(HDR) image generated based upon the chromatic sub-images, and whereincombining the chromatic sub-images into the HDR image comprises:identifying a saturated image value from a saturated sensor pixeloverlaid by a filter element of a first color filter type; and adjustingthe saturated image value based upon another image value extracted fromanother sensor pixel adjacent to the saturated sensor pixel but overlaidby another filter element of a second color filter type, wherein theother image value is determined with reference to a given wavelengthdependent efficiency of the second color filter type.
 18. The method ofclaim 15, wherein the CFA includes a first color filter typecorresponding to a first illumination color, a second color filter typecorresponding to a second illumination color, and a third color filtertype corresponding to a third illumination color.
 19. The method ofclaim 18, further comprising: extracting first color data correspondingto the first illumination color from a given sensor pixel overlaid by afilter element of the second color filter type with reference to awavelength dependent efficiency of the second color filter type.
 20. Themethod of claim 19, further comprising: optimizing the first color dataextracted from the given sensor pixel overlaid by the filter element ofthe second color filter type using a weighted combination of the firstcolor data from the given sensor pixel and other first color data outputfrom another sensor pixel adjacent to the given sensor pixel in thearray that is overlaid by another filter element of the first colorfilter type.
 21. The method of claim 18, wherein combining the chromaticsub-images into the composite color image comprises: scaling first imagevalues output from the sensor pixels under first filter elements of thefirst color filter type based at least in part upon a first wavelengthdependent efficiency of the first color filter type; scaling secondimage values output from the sensor pixels under second filter elementsof the second color filter type based at least in part upon a secondwavelength dependent efficiency of the second color filter type; andscaling third image values output from the sensor pixels under thirdfilter elements of the third color filter type based at least in partupon a third wavelength dependent efficiency of the third color filtertypes.
 22. The method of claim 14, further comprising: acquiring a fullcolor image with all of the illumination colors of the multi-chromaticilluminator enabled; and combining the composite color image with thefull color image with one or more convolution kernels that trade offimage characteristics to generate an optimized composite color image.23. The method of claim 22, wherein the one or more convolution kernelsare two-dimensional convolution kernels selected based upon solving amerit function that trades off the image characteristics including adynamic range and a spatial resolution.
 24. The method of claim 22,further comprising: temporally tracking an object moving in thechromatic sub-images; and adaptively adjusting the one or moreconvolution kernels as the object moves between a high noise pixel and alow noise pixel.
 25. The method of claim 14, further comprising:determining a total power of a given chromatic sub-image by summingimage values of the given chromatic sub-image; comparing the total powerto a threshold total power; and reacquiring the given chromaticsub-image when the total power is below the threshold total power.